Skip to content

Refactor Agents SDK instrumentation. #854

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dot-agi merged 73 commits into from
Mar 24, 2025
Merged

Refactor Agents SDK instrumentation. #854

dot-agi merged 73 commits into from
Mar 24, 2025

Conversation

@tcdent
Copy link
Contributor

@tcdent tcdent commented Mar 15, 2025 •
edited
Loading

Thus implementation demonstrates a sustainable pattern for all instrumentation that gets included in AgentOps.

  • Uses semantic conventions exclusively.
  • All pertinent data is exported, including workflow input/output, tool use, and completions.
  • Spans nest under the parent trace and are opened/closed in a lifecycle that is compatible with agent's built-in tracing module.
  • Tooling for generating test data from Response objects from openai-agents and completions & responses from openai
  • Example code that uses handoffs.
  • Better span_names
    • agents.function should be something like agents.tool.<tool_name>
    • agents.response should have the agent name
  • Agent span is closing early? Should include the tool and llm calls within it's start/end times.
  • Completion data is getting merged with the agent. We should have separate spans for the agent making the action and then the llm call as a child span. This appears to be a limitation of the way openai_agents hands off to tracing; lots of things get categorized as response, but it works much better now.
  • instrumentation.openai is incomplete. I created it to be able to instrument responses api calls, but ended up incorporating that into the instrumentation.openai_agents. This will be applied in a new PR.
  • Attributes have tests, but the exporter/processor does not.

devin-ai-integration bot and others added 17 commits March 14, 2025 23:07
@devin-ai-integration
Fix: Improve serialization of completions/responses in Agents SDK ins…
b248d24 
…trumentation

Co-Authored-By: [email protected] <[email protected]>
@devin-ai-integration
Fix: Improve serialization of completions/responses in Agents SDK ins…
30eb11e 
…trumentation

Co-Authored-By: [email protected] <[email protected]>
@tcdent
Tests for completions.
d6c2f8a
@tcdent
Separate OpenAI tests into completion and responses
9283b83
@tcdent
Refactor completions and responses unit tests.
770b37a
@tcdent
agents SDK test using semantic conventions.
29a115f
@tcdent
semantic conventions in openai completions and responses tests
a67deb7
@tcdent
Exporter refactor and generalization. standardization and simplificat…
6f1e77a 
…ion of version of values into semantic types.
@tcdent
Continued refactor of Agents instrumentor. Usurp third-party implemen…
5b4e940 
…tation.
@tcdent
Semantic conventions for messages.
0169502
@tcdent
Tools for generating real test data from OpenAI Agents.
960a01f
@tcdent
support tool calls and set of responses. missing import
124a469
@tcdent
reasoning tokens, semantic conventions, and implementation in OpenAI …
ce5b122 
…agent responses.
@tcdent
populate agents SDK tests with fixture data. Simplify fixture data ge…
039978b 
…neration tooling. increased test coverage
@tcdent
Add chat completion support to openai_agents. Cleanup OpenAI agents i…
1fa5fb6 
…nstrumentation.
@tcdent
Agents instrumentor cleanup.
72ab339
@tcdent
Cleanup.
d206b67
@tcdent tcdent requested a review from teocns March 15, 2025 09:05
dot-agi
dot-agi requested changes Mar 15, 2025
View reviewed changes
Copy link
Member

@dot-agi dot-agi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tcdent for the PR! This is helpful in moving towards a more stable instrumentation for the OpenAI provider and Agents framework.

I have requested some changes primarily on the semantic conventions being used to record the attributes. This would prevent duplication of data and provide an ordered structure of extracting data from queries in the backend.

@tcdent tcdent marked this pull request as draft March 15, 2025 17:40
@tcdent
Copy link
Contributor Author

tcdent commented Mar 15, 2025

@the-praxs Did you run this? Post your script and output below.

@dot-agi
Copy link
Member

dot-agi commented Mar 15, 2025

@the-praxs Did you run this? Post your script and output below.

https://gist.github.com/the-praxs/73fe232afb46c6cccb110dd4de9db0ba

@codecov
Copy link

codecov bot commented Mar 22, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 64.38172% with 265 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
agentops/instrumentation/openai_agents/exporter.py 46.73% 98 Missing ⚠️
...strumentation/openai_agents/attributes/response.py 36.08% 62 Missing ⚠️
...instrumentation/openai_agents/attributes/tokens.py 57.69% 55 Missing ⚠️
.../instrumentation/openai_agents/attributes/model.py 64.40% 21 Missing ⚠️
...rumentation/openai_agents/attributes/completion.py 87.34% 10 Missing ⚠️
...tops/instrumentation/openai_agents/instrumentor.py 84.09% 7 Missing ⚠️
...instrumentation/openai_agents/attributes/common.py 93.65% 4 Missing ⚠️
agentops/instrumentation/openai_agents/__init__.py 76.92% 3 Missing ⚠️
agentops/helpers/serialization.py 88.23% 2 Missing ⚠️
...gentops/instrumentation/openai_agents/processor.py 91.66% 2 Missing ⚠️
... and 1 more

📢 Thoughts on this report? Let us know!

@tcdent tcdent marked this pull request as ready for review March 24, 2025 19:51
@dot-agi dot-agi self-requested a review March 24, 2025 19:59
@dot-agi
Copy link
Member

dot-agi commented Mar 24, 2025

Testing results here -

https://gist.github.com/the-praxs/251d08f2aca7810759c3b954305f5850

@dot-agi
Copy link
Member

dot-agi commented Mar 24, 2025

Tested streaming with examples/agents-examples/basic/stream_text.py and it works as expected.

dot-agi
dot-agi approved these changes Mar 24, 2025
View reviewed changes
Copy link
Member

@dot-agi dot-agi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM AND YOLO

tcdent and others added 3 commits March 24, 2025 16:29
@tcdent
Log deeplink to trace on AgentOps dashboard. (#879)
cc14de9 
* Log deeplink to trace on AgentOps dashboard.

* Test coverage, type checking.

* Get app_url from config.

* Don't format trace_id in the URL as a UUID, just a hex string.
@tcdent
Merge branch 'main' into serialization-fix-test
be94e41
@dot-agi
Merge branch 'main' into serialization-fix-test
c8a6531
@dot-agi dot-agi merged commit c1a4015 into main Mar 24, 2025
7 of 10 checks passed
@dot-agi dot-agi deleted the serialization-fix-test branch March 24, 2025 23:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dot-agi dot-agi dot-agi approved these changes

@teocns teocns Awaiting requested review from teocns

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Update Agents SDK to correctly export spans

3 participants

@tcdent @dot-agi